Feature - add missing jvm thread metrics#17135
Feature - add missing jvm thread metrics#17135JackN5y wants to merge 3 commits intoopen-telemetry:mainfrom
Conversation
Add experimental JVM thread metrics that are available in JMX Exporter but were missing from OpenTelemetry JMX metrics: - jvm.thread.daemon.count: Number of daemon threads currently running - jvm.thread.peak.count: Peak thread count since JVM start - jvm.thread.started.count: Total threads started since JVM start These metrics map to the following JMX Exporter equivalents: - jvm_threads_daemon - jvm_threads_peak - jvm_threads_started_total Note: Thread deadlock metrics (jvm_threads_deadlocked, jvm_threads_deadlocked_monitor) cannot be implemented as they require invoking JMX operations rather than reading attributes.
Add ExperimentalThreads class that exposes two new experimental metrics for JVM thread deadlock detection: - jvm.thread.deadlock.count: Number of platform threads in deadlock waiting to acquire object monitors or ownable synchronizers. Uses ThreadMXBean.findDeadlockedThreads() which detects deadlocks involving both object monitor locks and ownable synchronizers (e.g. ReentrantLock). - jvm.thread.monitor_deadlock.count: Number of platform threads in deadlock waiting to acquire object monitors only. Uses ThreadMXBean.findMonitorDeadlockedThreads() which detects deadlocks involving only object monitor locks (synchronized blocks). These metrics are equivalent to Prometheus client_java's jvm_threads_deadlocked and jvm_threads_deadlocked_monitor gauges. They cannot be implemented via JMX YAML rules because deadlock detection requires invoking MBean operations (findDeadlockedThreads, findMonitorDeadlockedThreads), not reading MBean attributes. The metrics are registered as experimental and are emitted only when experimental telemetry is enabled via: -Dotel.instrumentation.runtime-telemetry.emit-experimental-telemetry=true In Prometheus format they will appear as: jvm_thread_deadlock_count jvm_thread_monitor_deadlock_count
SylvainJuge
left a comment
There was a problem hiding this comment.
JVM runtime metrics are part of the JVM runtime semantic conventions, so any addition or change should also be included in https://github.com/open-telemetry/semantic-conventions/tree/main/model/jvm
The usual process is thus in a few steps:
- open a draft PR in semconv
- open a draft PR in instrumentation to provide implementation
- then handle any feedback on semconv and implementation until they are ready for merge
Also, if you add new metrics, there are actually two implementations that need to be updated:
- one in the
jvm.yamlfor use withjmx-scraper(in contrib) from outside of the JVM - one in
runtime-telemetrywhich is used by instrumentation
| DaemonThreadCount: | ||
| metric: daemon.count | ||
| type: updowncounter | ||
| unit: "{thread}" | ||
| desc: Number of daemon threads currently running. |
There was a problem hiding this comment.
this is already covered with jvm.thread.count metric with the [jvm.thread.daemon](https://opentelemetry.io/docs/specs/semconv/registry/attributes/jvm/) boolean attribute.
Unfortulately, the daemon status can't be accessed through the JMX interface, however it is provided when the metrics are captured with runtime-telemetry module.
|
This PR has been labeled as stale due to lack of activity and needing author feedback. It will be automatically closed if there is no further activity over the next 7 days. |
No description provided.